{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/tomasonjo/blogs/blob/master/Star_Wars/Star%20Wars.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "q1rZf4Ucy3LO"
   },
   "source": [
    "* Updated to GDS 2.0 version\n",
    "* Link to original blog post: https://towardsdatascience.com/presenting-multiple-node-label-support-and-graph-mutability-features-of-the-neo4j-graph-data-a0b0ea744884\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "06L4oIQdynjr",
    "outputId": "046c9879-5980-4f7a-9e4d-5c050a9797b3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting neo4j\n",
      "  Downloading neo4j-4.4.2.tar.gz (89 kB)\n",
      "\u001b[?25l\r",
      "\u001b[K     |███▋                            | 10 kB 19.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███████▎                        | 20 kB 22.0 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████                     | 30 kB 24.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████▋                 | 40 kB 25.0 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████▎             | 51 kB 19.4 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████          | 61 kB 22.0 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▋      | 71 kB 22.0 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▎  | 81 kB 21.4 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████████| 89 kB 6.4 MB/s \n",
      "\u001b[?25hRequirement already satisfied: pytz in /usr/local/lib/python3.7/dist-packages (from neo4j) (2018.9)\n",
      "Building wheels for collected packages: neo4j\n",
      "  Building wheel for neo4j (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for neo4j: filename=neo4j-4.4.2-py3-none-any.whl size=115365 sha256=36cef6fe279d5c2a4f810a551b2d35752f3910dd8b661b402272ecca746e123c\n",
      "  Stored in directory: /root/.cache/pip/wheels/10/d6/28/95029d7f69690dbc3b93e4933197357987de34fbd44b50a0e4\n",
      "Successfully built neo4j\n",
      "Installing collected packages: neo4j\n",
      "Successfully installed neo4j-4.4.2\n"
     ]
    }
   ],
   "source": [
    "!pip install neo4j"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "iO4gYjzEylo1"
   },
   "outputs": [],
   "source": [
    "# Define Neo4j connections\n",
    "from neo4j import GraphDatabase\n",
    "host = 'bolt://3.231.25.240:7687'\n",
    "user = 'neo4j'\n",
    "password = 'hatchets-visitor-axes'\n",
    "driver = GraphDatabase.driver(host,auth=(user, password))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "-rJrMIHvylo4"
   },
   "outputs": [],
   "source": [
    "def drop_graph(name):\n",
    "    with driver.session() as session:\n",
    "        drop_graph_query = \"\"\"\n",
    "        CALL gds.graph.drop('{}');\n",
    "        \"\"\".format(name)\n",
    "        session.run(drop_graph_query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "z224eGE3ylo5"
   },
   "outputs": [],
   "source": [
    "# Import libraries\n",
    "import pandas as pd\n",
    "\n",
    "def read_query(query, params={}):\n",
    "    with driver.session() as session:\n",
    "        result = session.run(query, params)\n",
    "        return pd.DataFrame([r.values() for r in result], columns=result.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PW4r4r5lylo6"
   },
   "source": [
    "# Presenting multiple node label support and graph mutability features of the Neo4j Graph Data Science in the Star Wars universe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kvzewnySylo7"
   },
   "source": [
    "Just recently, the v1.1.0 version of the Neo4j Graph Data Science was released. I decided to take a short break from watching the Joy of Painting by Bob Ross, which I highly recommend, and inspect some of the new features. For me, the most anticipated new feature is the ability to filter nodes by labels when executing graph algorithms. Similarly to relationships, where we can choose which ones should be considered by the graph algorithm with the relationshipTypes parameter, we can now filter which nodes should be used with the new nodeLabels parameter. We will also take a look at the new projected graph mutability feature.\n",
    "\n",
    "I like to search for new networks to not get boring and repetitive in my blog posts. Here we will delve into the Star Wars world. The interactions between characters are available on GitHub, thanks to Evelina Gabasova. We will combine it with the dataset made available by Florent Georges, who scraped the Star Wars API and added descriptions from Wikipedia."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PzWJKe4iylo8"
   },
   "source": [
    "## Graph schema and import\n",
    "\n",
    "Our graph consists of characters that have INTERACTX relationships with other characters. The value of the X indicates in which episode the interaction occurred, e.g. INTERACTS1 indicates an interaction in the first episode. Each character also belongs to a single species, which is represented as a relationship to the species node.\n",
    "\n",
    "We will start by importing the interaction social network. The dataset is a bit tricky as characters don't have a unique id available throughout the episodes. Instead, in each episode, a new id is determined as a zero-based index of the character in the \"nodes\" element of the JSON file. To mitigate this issue, we use the apoc.cypher.runMany procedure to run two transactions for each episode. In the first, we store the nodes and assign them the value based on the zero-based index. In the second transaction, we use that value to store links between characters. This procedure is repeated for each episode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 927
    },
    "id": "VTkWnEDNylo9",
    "outputId": "8a01116c-56b1-4a08-ab6b-f6757751c58e"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>result</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>{''done'': 'done'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>{'constraintsRemoved': 0, 'indexesRemoved': 0,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>{''done'': 'done'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>{'constraintsRemoved': 0, 'indexesRemoved': 0,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>{''done'': 'done'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>{'constraintsRemoved': 0, 'indexesRemoved': 0,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>{''done'': 'done'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>{'constraintsRemoved': 0, 'indexesRemoved': 0,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>{''done'': 'done'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>{'constraintsRemoved': 0, 'indexesRemoved': 0,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>{''done'': 'done'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>{'constraintsRemoved': 0, 'indexesRemoved': 0,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>{''done'': 'done'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>{'constraintsRemoved': 0, 'indexesRemoved': 0,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>{''done'': 'done'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>{'constraintsRemoved': 0, 'indexesRemoved': 0,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>{''done'': 'done'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>{'constraintsRemoved': 0, 'indexesRemoved': 0,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>{''done'': 'done'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>{'constraintsRemoved': 0, 'indexesRemoved': 0,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>{''done'': 'done'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>{'constraintsRemoved': 0, 'indexesRemoved': 0,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>{''done'': 'done'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>{'constraintsRemoved': 0, 'indexesRemoved': 0,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>{''done'': 'done'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>{'constraintsRemoved': 0, 'indexesRemoved': 0,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>{''done'': 'done'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>{'constraintsRemoved': 0, 'indexesRemoved': 0,...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               result\n",
       "0                                  {''done'': 'done'}\n",
       "1   {'constraintsRemoved': 0, 'indexesRemoved': 0,...\n",
       "2                                  {''done'': 'done'}\n",
       "3   {'constraintsRemoved': 0, 'indexesRemoved': 0,...\n",
       "4                                  {''done'': 'done'}\n",
       "5   {'constraintsRemoved': 0, 'indexesRemoved': 0,...\n",
       "6                                  {''done'': 'done'}\n",
       "7   {'constraintsRemoved': 0, 'indexesRemoved': 0,...\n",
       "8                                  {''done'': 'done'}\n",
       "9   {'constraintsRemoved': 0, 'indexesRemoved': 0,...\n",
       "10                                 {''done'': 'done'}\n",
       "11  {'constraintsRemoved': 0, 'indexesRemoved': 0,...\n",
       "12                                 {''done'': 'done'}\n",
       "13  {'constraintsRemoved': 0, 'indexesRemoved': 0,...\n",
       "14                                 {''done'': 'done'}\n",
       "15  {'constraintsRemoved': 0, 'indexesRemoved': 0,...\n",
       "16                                 {''done'': 'done'}\n",
       "17  {'constraintsRemoved': 0, 'indexesRemoved': 0,...\n",
       "18                                 {''done'': 'done'}\n",
       "19  {'constraintsRemoved': 0, 'indexesRemoved': 0,...\n",
       "20                                 {''done'': 'done'}\n",
       "21  {'constraintsRemoved': 0, 'indexesRemoved': 0,...\n",
       "22                                 {''done'': 'done'}\n",
       "23  {'constraintsRemoved': 0, 'indexesRemoved': 0,...\n",
       "24                                 {''done'': 'done'}\n",
       "25  {'constraintsRemoved': 0, 'indexesRemoved': 0,...\n",
       "26                                 {''done'': 'done'}\n",
       "27  {'constraintsRemoved': 0, 'indexesRemoved': 0,..."
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import_interactions = \"\"\"\n",
    "\n",
    "UNWIND range(1,7) as episode \n",
    "CALL apoc.load.json('https://raw.githubusercontent.com/evelinag/StarWars-social-network/master/networks/starwars-episode-' + toString(episode) + '-interactions.json')\n",
    "YIELD value as data \n",
    "CALL apoc.cypher.runMany(\"\n",
    "    WITH $data.nodes as nodes \n",
    "    UNWIND range(0,size(nodes)-1) as value \n",
    "    MERGE (c:Character{id:nodes[value].name}) \n",
    "    SET c.value = value RETURN distinct 'done'\n",
    "    ;\n",
    "    WITH $data as data, $episode as episode \n",
    "    UNWIND data.links as link \n",
    "    MERGE (source:Character{value:link.source}) \n",
    "    MERGE (target:Character{value:link.target}) \n",
    "    WITH source, target, link.value as weight, episode \n",
    "    CALL apoc.create.relationship(source,'INTERACTS' + episode, {weight:weight}, target) YIELD rel \n",
    "    RETURN distinct 'done'\", {data:data, episode:episode},{statistics:true,timeout:10}) \n",
    "    YIELD result RETURN result\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "read_query(import_interactions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vAXACUbNylo_"
   },
   "source": [
    "We will continue by enriching our graph with information about the species. Unfortunately, the names of characters between the two datasets do not match exactly. There are several approaches that we could use to find close matches of the characters' names. APOC library has a whole section dedicated to text-similarity procedures. I decided to use the Full-text search index to match the characters' names. We begin by defining the full-text search index on the id property of characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 49
    },
    "id": "Ou2Iv5I6ylo_",
    "outputId": "02350625-5aa1-4461-f6b1-72b6e645080d"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "define_fts = \"\"\"\n",
    "CREATE FULLTEXT INDEX names IF NOT EXISTS FOR (c:Character) ON EACH [c.id]\n",
    "\"\"\"\n",
    "read_query(define_fts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b6DvvuRyylpA"
   },
   "source": [
    "Each full-text search query might return more than a single result, but we will look only at the top result. If the score of the top result is above an arbitrary threshold, 0.85 in our case, we will assume it is the same character. We can look at some examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "2QIUNkSyylpA",
    "outputId": "ea2e5166-1320-4d20-948a-4f4bdfe8b663"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>species_dataset_name</th>\n",
       "      <th>interactions_dataset_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Luke Skywalker</td>\n",
       "      <td>LUKE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>C-3PO</td>\n",
       "      <td>C-3PO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Darth Vader</td>\n",
       "      <td>DARTH VADER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Leia Organa</td>\n",
       "      <td>LEIA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Owen Lars</td>\n",
       "      <td>OWEN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  species_dataset_name interactions_dataset_name\n",
       "0       Luke Skywalker                      LUKE\n",
       "1                C-3PO                     C-3PO\n",
       "2          Darth Vader               DARTH VADER\n",
       "3          Leia Organa                      LEIA\n",
       "4            Owen Lars                      OWEN"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_fts = \"\"\"\n",
    "\n",
    "CALL apoc.load.json('https://raw.githubusercontent.com/fgeorges/star-wars-dataset/master/data/enriched.json')\n",
    "YIELD value\n",
    "UNWIND value.people as person\n",
    "CALL db.index.fulltext.queryNodes(\"names\", replace(person.name,'é','e'))\n",
    "YIELD node,score \n",
    "RETURN person.name as species_dataset_name, \n",
    "       collect(node)[0].id as interactions_dataset_name\n",
    "LIMIT 5\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "read_query(example_fts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "66j4_fiZylpB"
   },
   "source": [
    "More or less, it seems that the species dataset contains full names, while the interactions dataset contains only the first names of the characters. As the full-text search matching seems fine, let us proceed and store the additional information to the graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 49
    },
    "id": "0fPHHNBoylpB",
    "outputId": "5f59ecdc-6734-49e1-9cfa-6c89d6d129af"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "store_species_fts = \"\"\"\n",
    "\n",
    "CALL apoc.load.json('https://raw.githubusercontent.com/fgeorges/star-wars-dataset/master/data/enriched.json')\n",
    "YIELD value\n",
    "UNWIND value.people as person \n",
    "// search for characters \n",
    "CALL db.index.fulltext.queryNodes(\"names\", replace(person.name,'é','e')) \n",
    "YIELD node,score \n",
    "// collect the top hit \n",
    "WITH person, collect([node,score])[0] as top_hit \n",
    "WITH person, top_hit[0] as node, top_hit[1] as score \n",
    "// threshold \n",
    "WHERE score > 0.85 \n",
    "// enrich characters \n",
    "SET node += apoc.map.clean(person, ['films','vehicles','starships','species'],['n/a']) \n",
    "WITH node, person.species as person_species \n",
    "UNWIND person_species as species \n",
    "MERGE (s:Species{id:species}) \n",
    "MERGE (node)-[:BELONG_TO]->(s)\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "read_query(store_species_fts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XKq2cUjQylpB"
   },
   "source": [
    "We will also import additional meta-data about the species."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 49
    },
    "id": "AIpZYBYWylpC",
    "outputId": "6ca0aa5b-4b80-4d14-ceba-874f42a8160e"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "species_meta = \"\"\"\n",
    "\n",
    "CALL apoc.load.json('https://raw.githubusercontent.com/fgeorges/star-wars-dataset/master/data/enriched.json')\n",
    "YIELD value \n",
    "UNWIND value.species as species\n",
    "MATCH (s:Species{id:species.url})\n",
    "SET s += apoc.map.clean(species, ['films','homeworld','people','url'],['n/a','unknown'])\n",
    "\n",
    "\"\"\"\n",
    "read_query(species_meta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nfpJHtgVylpC"
   },
   "source": [
    "Unfortunately, we haven't assigned the species to all of our characters yet. Some characters show up in the first dataset that do not show up in the second. As the focus of this blog post is not to show how to scrape the internet efficiently, I prepared a CSV file that we can use to assign missing species values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 49
    },
    "id": "Isk0fwI_ylpC",
    "outputId": "a69e2bb7-03d9-4426-a66b-84a768cd7869"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "store_species = \"\"\"\n",
    "\n",
    "LOAD CSV WITH HEADERS FROM \"https://raw.githubusercontent.com/tomasonjo/blogs/master/Star_Wars/star_wars_species.csv\" as row \n",
    "MATCH (c:Character{id:row.name}) \n",
    "MERGE (s:Species{name:row.species})\n",
    "MERGE (c)-[:BELONG_TO]->(s)\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "read_query(store_species)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nepI-Y-_ylpD"
   },
   "source": [
    "Some species have only one or two members, so I decided to group them under \"Other\" species. This way, we will make our further graph analysis more relevant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 49
    },
    "id": "tgAz8pv5ylpD",
    "outputId": "5d0523c0-b691-4418-e04f-992dec824cef"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "group_species = \"\"\"\n",
    "\n",
    "MATCH (s:Species)\n",
    "WHERE count{ (s)<-[:BELONG_TO]-() } <= 2\n",
    "MERGE (unknown:Species{name:'Other'}) \n",
    "WITH s, unknown \n",
    "MATCH (s)<-[:BELONG_TO]-(character)\n",
    "MERGE (unknown)<-[:BELONG_TO]-(character) \n",
    "DETACH DELETE s\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "read_query(group_species)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FTWpBDSZylpD"
   },
   "source": [
    "## Multiple node label support\n",
    "\n",
    "We can now look at the multiple node label projection and how it allows us to filter nodes when executing graph algorithms. As we will be using the native projection, we have to assign secondary labels to characters based on their species. We will use the <code>apoc.create.addLabels</code> procedure to assign secondary labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 81
    },
    "id": "iEnNxccMylpE",
    "outputId": "2e9e0080-43eb-4d99-de11-ee77b982d41c"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>'done'</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>done</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  'done'\n",
       "0   done"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "secondary_label = \"\"\"\n",
    "\n",
    "MATCH (c:Character)-[:BELONG_TO]->(species)\n",
    "CALL apoc.create.addLabels([id(c)], [species.name]) YIELD node\n",
    "RETURN distinct 'done'\n",
    "\n",
    "\"\"\"\n",
    "read_query(secondary_label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xEfhSaTmylpE"
   },
   "source": [
    "If you are new to the concept of the GDS graph catalog, you might want to look at one of my previous blog posts. We will use the array option to describe the node labels we want to project. Nodes belonging to all five species will be projected. All relationships in the interaction network will also be projected and treated as undirected and weighted. As there could be more than a single relationship between a given pair of nodes, we are projecting a multigraph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 81
    },
    "id": "rKWMyA-5ylpE",
    "outputId": "ef6238fd-86bb-404d-fda0-74fcab72ea22"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nodeProjection</th>\n",
       "      <th>relationshipProjection</th>\n",
       "      <th>graphName</th>\n",
       "      <th>nodeCount</th>\n",
       "      <th>relationshipCount</th>\n",
       "      <th>projectMillis</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>{'Human': {'label': 'Human', 'properties': {}}...</td>\n",
       "      <td>{'INTERACTS5': {'orientation': 'UNDIRECTED', '...</td>\n",
       "      <td>starwars</td>\n",
       "      <td>110</td>\n",
       "      <td>3506</td>\n",
       "      <td>186</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      nodeProjection  \\\n",
       "0  {'Human': {'label': 'Human', 'properties': {}}...   \n",
       "\n",
       "                              relationshipProjection graphName  nodeCount  \\\n",
       "0  {'INTERACTS5': {'orientation': 'UNDIRECTED', '...  starwars        110   \n",
       "\n",
       "   relationshipCount  projectMillis  \n",
       "0               3506            186  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "native_gds = \"\"\"\n",
    "\n",
    "CALL gds.graph.project('starwars',\n",
    "    ['Other','Human','Droid','Neimodian','Gungan'],\n",
    "    {INTERACTS1:{type:'INTERACTS1', orientation:'UNDIRECTED', properties:['weight']},\n",
    "     INTERACTS2:{type:'INTERACTS2', orientation:'UNDIRECTED', properties:['weight']},\n",
    "     INTERACTS3:{type:'INTERACTS3', orientation:'UNDIRECTED', properties:['weight']},\n",
    "     INTERACTS4:{type:'INTERACTS4', orientation:'UNDIRECTED', properties:['weight']},\n",
    "     INTERACTS5:{type:'INTERACTS5', orientation:'UNDIRECTED', properties:['weight']},\n",
    "     INTERACTS6:{type:'INTERACTS6', orientation:'UNDIRECTED', properties:['weight']}})\n",
    "\n",
    "\"\"\"\n",
    "read_query(native_gds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4a5I90hEylpF"
   },
   "source": [
    "To start off, let's compute the weighted PageRank on the whole projected graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "_GRekIUxylpF",
    "outputId": "6f5fd757-1cdf-45d8-f4b9-38d28efb37df"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Han Solo</td>\n",
       "      <td>3.682756</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Anakin Skywalker</td>\n",
       "      <td>3.555554</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Padmé Amidala</td>\n",
       "      <td>3.394732</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>C-3PO</td>\n",
       "      <td>3.337645</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Luke Skywalker</td>\n",
       "      <td>3.293567</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               name     score\n",
       "0          Han Solo  3.682756\n",
       "1  Anakin Skywalker  3.555554\n",
       "2     Padmé Amidala  3.394732\n",
       "3             C-3PO  3.337645\n",
       "4    Luke Skywalker  3.293567"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "whole_pagerank = \"\"\"\n",
    "\n",
    "CALL gds.pageRank.stream('starwars',\n",
    "    {relationshipWeightProperty:'weight'})\n",
    "YIELD nodeId, score\n",
    "RETURN gds.util.asNode(nodeId).name as name, score\n",
    "ORDER BY score DESC LIMIT 5\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "read_query(whole_pagerank)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "C67t8XdNylpF"
   },
   "source": [
    "The results are not surprising, except maybe that Han Solo seems to be the most important character in the galaxy. This is due to Anakin Skywalker and Darth Vader being treated as two separate entities in our network, otherwise probably Anakin would be on top. Nonetheless, judging by the PageRank score, all the top five characters are quite similar in their importance, and no one really stands out.\n",
    "\n",
    "Let's say we want to find the most important characters in the Gungan social network. We can consider only Gungans in the algorithm with the nodeLabels parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 175
    },
    "id": "O5RBXw4rylpF",
    "outputId": "8a097b55-aeeb-4ccc-c7a6-347fc7059a32"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>JAR JAR</td>\n",
       "      <td>1.359341</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>TARPALS</td>\n",
       "      <td>1.053097</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>BOSS NASS</td>\n",
       "      <td>1.053097</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>GENERAL CEEL</td>\n",
       "      <td>0.379426</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           name     score\n",
       "0       JAR JAR  1.359341\n",
       "1       TARPALS  1.053097\n",
       "2     BOSS NASS  1.053097\n",
       "3  GENERAL CEEL  0.379426"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gungan_pagerank = \"\"\"\n",
    "\n",
    "CALL gds.pageRank.stream('starwars',\n",
    "    {nodeLabels:['Gungan'], relationshipWeightProperty:'weight'})\n",
    "YIELD nodeId, score \n",
    "RETURN gds.util.asNode(nodeId).id as name, score \n",
    "ORDER BY score DESC LIMIT 5\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "read_query(gungan_pagerank)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vUX0bifZylpG"
   },
   "source": [
    "As you might expect, the man, the legend Jar Jar, comes out on top.\n",
    "\n",
    "We can also combine both node and relationship filters. We will compute PageRank based on the network of humans and other species interactions in the fifth and sixth episodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "L2iTCftQylpG",
    "outputId": "6b4ad91d-e744-446b-d822-0fefee19f3f4"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Han Solo</td>\n",
       "      <td>3.899221</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Leia Organa</td>\n",
       "      <td>2.768740</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Luke Skywalker</td>\n",
       "      <td>2.517664</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Padmé Amidala</td>\n",
       "      <td>1.988432</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Biggs Darklighter</td>\n",
       "      <td>1.947932</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                name     score\n",
       "0           Han Solo  3.899221\n",
       "1        Leia Organa  2.768740\n",
       "2     Luke Skywalker  2.517664\n",
       "3      Padmé Amidala  1.988432\n",
       "4  Biggs Darklighter  1.947932"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "node_and_rel_filter_pagerank = \"\"\"\n",
    "\n",
    "CALL gds.pageRank.stream('starwars',\n",
    "    {nodeLabels:['Other','Human'], relationshipTypes:['INTERACTS5','INTERACTS6'], \n",
    "    relationshipWeightProperty:'weight'}) \n",
    "YIELD nodeId, score \n",
    "RETURN gds.util.asNode(nodeId).name as name, score\n",
    "ORDER BY score DESC LIMIT 5\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "read_query(node_and_rel_filter_pagerank)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SERU0mq_ylpG"
   },
   "source": [
    "It looks like the kids Luke and Leia have grown up. The addition of the node filtering feature allows us to be very precise when describing the subset of the projected graph we want to consider as an input of a graph algorithm. After we are done with the analysis, we can drop the projected graph from the catalog."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "KAwPZzmZylpG"
   },
   "outputs": [],
   "source": [
    "drop_graph('starwars')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EMJCoR-8ylpH"
   },
   "source": [
    "## Graph mutability\n",
    "\n",
    "With the graph mutability feature, graph algorithms have gotten a third mode mutate on top of the previously supported stream and write. We can look at the official documentation to get a better sense of what it offers:\n",
    "\n",
    ">The mutate mode is very similar to the write mode but instead of writing results to the Neo4j database, they are >made available at the in-memory graph. Note that the mutateProperty must not exist in the in-memory graph >beforehand. This enables running multiple algorithms on the same in-memory graph without writing results to Neo4j >in-between algorithm executions.\n",
    "\n",
    "Basically, mutate mode is almost the same as the write mode, except it doesn't write results directly to the Neo4j stored graph, but instead stores the results to the same in-memory projected graph, that was used as an input of the graph algorithm. Later, if we want to, we can still store the mutated properties of the in-memory graph to Neo4j with the gds.graph.writeNodeProperties procedure.\n",
    "\n",
    "We will start by projecting a new in-memory graph. This time we will use cypher projection. In the node query, we will filter only characters that appeared in the first episode. We also provide the labels column, which will allow us to filter nodes when executing algorithms. If you look closely, the labels column is in the form of an array, which means that a single node can have multiple labels. Also, with cypher projection, we can provide a virtual label, as shown in the query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 81
    },
    "id": "wecvx-0qylpH",
    "outputId": "01915e02-caee-461a-a6e6-dfe8fe7d6c15"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nodeQuery</th>\n",
       "      <th>relationshipQuery</th>\n",
       "      <th>graphName</th>\n",
       "      <th>nodeCount</th>\n",
       "      <th>relationshipCount</th>\n",
       "      <th>projectMillis</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>MATCH (species)&lt;-[:BELONG_TO]-(c:Character) \\n...</td>\n",
       "      <td>MATCH (c1:Character)-[r]-(c2:Character) \\n    ...</td>\n",
       "      <td>starwars_cypher</td>\n",
       "      <td>37</td>\n",
       "      <td>1098</td>\n",
       "      <td>197</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           nodeQuery  \\\n",
       "0  MATCH (species)<-[:BELONG_TO]-(c:Character) \\n...   \n",
       "\n",
       "                                   relationshipQuery        graphName  \\\n",
       "0  MATCH (c1:Character)-[r]-(c2:Character) \\n    ...  starwars_cypher   \n",
       "\n",
       "   nodeCount  relationshipCount  projectMillis  \n",
       "0         37               1098            197  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gds_cypher = \"\"\"\n",
    "\n",
    "CALL gds.graph.project.cypher('starwars_cypher', \n",
    "    'MATCH (species)<-[:BELONG_TO]-(c:Character) \n",
    "    // filter characters from first episode \n",
    "    WHERE (c)-[:INTERACTS1]-() \n",
    "    RETURN id(c) as id, [species.name] as labels', \n",
    "    'MATCH (c1:Character)-[r]-(c2:Character) \n",
    "    RETURN id(c1) as source, id(c2) as target, type(r) as type, r.weight as weight', \n",
    "    {validateRelationships:false} )\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "read_query(gds_cypher)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vHlmS1xpylpH"
   },
   "source": [
    "Another new feature of the 1.1 release is the validateRelationships parameter of the cypher projection. The default setting throws an error if we try to project relationships between nodes that are not described in the node query. As we project only characters from the first episode in the node query but try to project interactions between all characters in the relationship query, we would get an error. We could either add a filter in the relationship query or just set the validateRelationship parameter to false. This way, all relationships between nodes not described in the node query will be silently ignored during the graph projection.\n",
    "\n",
    "Finally, we can try out the mutate mode of the Louvain algorithm, which is used for community detection. We will investigate the community structure of the first episode within species and not on the whole network. To achieve this, we have to run the algorithm separately for each species and consider only characters from the given species with the nodeLabels parameter and only interactions from the first episode with the relationshipTypes parameter.  To define the designated property used to store results to the in-memory graph, we use the mutateProperty parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "_jgM48K8ylpH",
    "outputId": "f14bb977-3b6d-4732-9d88-53632bea4fb5"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>species</th>\n",
       "      <th>communityCount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Human</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Droid</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Neimodian</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Gungan</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Other</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     species  communityCount\n",
       "0      Human               4\n",
       "1      Droid               2\n",
       "2  Neimodian               1\n",
       "3     Gungan               1\n",
       "4      Other               5"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mutate_first_episode = \"\"\"\n",
    "\n",
    "MATCH (s:Species)\n",
    "CALL gds.louvain.mutate('starwars_cypher',\n",
    "    {nodeLabels:[s.name], mutateProperty: s.name + '_1', relationshipTypes:['INTERACTS1']}) \n",
    "YIELD communityCount \n",
    "RETURN s.name as species, communityCount\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "read_query(mutate_first_episode)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "h2uSz03QylpI"
   },
   "source": [
    "We can now use the mutated properties of the characters' community from the first episode as seed property for calculating communities in the second episode. Louvain algorithm is initialized by assigning each node a unique id. With the seedProperty parameter, we can define the initial unique id of each node. If you want to learn more about the seed property and its use-cases, I wrote a blog post about it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "WbkS_ZMUylpI",
    "outputId": "11e6dd16-dc75-4a69-abfe-2f53b6590689"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>species</th>\n",
       "      <th>communityCount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Human</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Droid</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Neimodian</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Gungan</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Other</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     species  communityCount\n",
       "0      Human               2\n",
       "1      Droid               2\n",
       "2  Neimodian               1\n",
       "3     Gungan               1\n",
       "4      Other               3"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mutate_second_episode = \"\"\"\n",
    "\n",
    "MATCH (s:Species) \n",
    "CALL gds.louvain.mutate('starwars_cypher', \n",
    "    {nodeLabels:[s.name], mutateProperty: s.name + '_2',\n",
    "     relationshipTypes:['INTERACTS2'], seedProperty: s.name + '_1'}) \n",
    "YIELD communityCount \n",
    "RETURN s.name as species, communityCount\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "read_query(mutate_second_episode)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XnWPGlTUylpI"
   },
   "source": [
    "We can observe that the count of communities in the second episode has slightly reduced. To write the mutated properties back to Neo4j stored graph we can use the\n",
    "gds.graph.writeNodeProperties procedure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 81
    },
    "id": "TxFKcCrgylpI",
    "outputId": "12f19988-aca3-4236-fe0b-27bce4750689"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>writeMillis</th>\n",
       "      <th>graphName</th>\n",
       "      <th>nodeProperties</th>\n",
       "      <th>propertiesWritten</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>13</td>\n",
       "      <td>starwars_cypher</td>\n",
       "      <td>[Human_1, Human_2]</td>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   writeMillis        graphName      nodeProperties  propertiesWritten\n",
       "0           13  starwars_cypher  [Human_1, Human_2]                 34"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "write_mutated_properties = \"\"\"\n",
    "\n",
    "CALL gds.graph.writeNodeProperties('starwars_cypher', \n",
    "    ['Human_1','Human_2'])\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "read_query(write_mutated_properties)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "je6IAdGnylpJ"
   },
   "source": [
    "We can now examine the community structure of humans within the first episode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 175
    },
    "id": "SzTtUvrXylpJ",
    "outputId": "ade52876-99f6-4497-f622-a2a7350e32b2"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>community</th>\n",
       "      <th>size</th>\n",
       "      <th>members</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9</td>\n",
       "      <td>5</td>\n",
       "      <td>[QUI-GON, PADME, JIRA, SHMI, KITSTER]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>[OBI-WAN, CAPTAIN PANAKA, SIO BIBBLE]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11</td>\n",
       "      <td>3</td>\n",
       "      <td>[EMPEROR, VALORUM, BAIL ORGANA]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>12</td>\n",
       "      <td>6</td>\n",
       "      <td>[RIC OLIE, ANAKIN, MACE WINDU, RABE, BRAVO TWO...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   community  size                                            members\n",
       "0          9     5              [QUI-GON, PADME, JIRA, SHMI, KITSTER]\n",
       "1          4     3              [OBI-WAN, CAPTAIN PANAKA, SIO BIBBLE]\n",
       "2         11     3                    [EMPEROR, VALORUM, BAIL ORGANA]\n",
       "3         12     6  [RIC OLIE, ANAKIN, MACE WINDU, RABE, BRAVO TWO..."
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_first_episode = \"\"\"\n",
    "\n",
    "MATCH (c:Human) \n",
    "WHERE c.Human_1 IS NOT NULL \n",
    "RETURN c.Human_1 as community,\n",
    "       count(*) as size,\n",
    "       collect(c.id) as members\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "read_query(results_first_episode)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_e-q3YpXylpJ"
   },
   "source": [
    "And inspect how those communities evolved in the second episode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 112
    },
    "id": "ssjqeZ-2ylpJ",
    "outputId": "1c985e67-3947-4e9a-f4d0-cc824ec185c3"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>community</th>\n",
       "      <th>size</th>\n",
       "      <th>members</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11</td>\n",
       "      <td>7</td>\n",
       "      <td>[QUI-GON, OBI-WAN, EMPEROR, CAPTAIN PANAKA, JI...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>12</td>\n",
       "      <td>10</td>\n",
       "      <td>[SIO BIBBLE, PADME, RIC OLIE, ANAKIN, SHMI, KI...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   community  size                                            members\n",
       "0         11     7  [QUI-GON, OBI-WAN, EMPEROR, CAPTAIN PANAKA, JI...\n",
       "1         12    10  [SIO BIBBLE, PADME, RIC OLIE, ANAKIN, SHMI, KI..."
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_second_episode = \"\"\"\n",
    "\n",
    "MATCH (c:Human)\n",
    "WHERE c.Human_2 IS NOT NULL\n",
    "RETURN c.Human_2 as community,\n",
    "       count(*) as size,\n",
    "       collect(c.id) as members\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "read_query(results_second_episode)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qIpmgbdOylpJ"
   },
   "source": [
    "Unfortunately, I am not a domain expert to give some useful insights about the community structure.\n",
    "\n",
    "\n",
    "## Conclusion\n",
    "\n",
    "With multiple node label support, we are now able to project the whole graph in memory and be very specific in describing the subgraph we want to use as an input to the graph algorithm as we can now filter both nodes and relationships. Graph mutability allows us to chain various graph algorithms without the need to drop and recreate projected graphs with newly computed properties. In my opinion, this is a great addition to the Neo4j Graph Data Science library.\n",
    "\n",
    "\n",
    "If you have any cool dataset in mind you would like me to analyze or have any feedback as to what you would like to see next, let me know."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2qMIp1GDylpJ"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "include_colab_link": true,
   "name": "Star Wars.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
